Redundant system and redundancy method

ABSTRACT

A redundant system includes a primary system including a first node and a second node, and a secondary system including a third node and a fourth node. When the secondary system in place of the primary system operates, the fourth node executes first takeover processing or second takeover processing, the first takeover processing taking over the primary system on the basis of data update information acquired from either a second inter-system transfer path or a second intra-system transfer path, and the second takeover processing taking over the primary system on the basis of both the data update information acquired from the second inter-system transfer path and the data update information acquired from the second intra-system transfer path.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2014-127732, filed on Jun. 20,2014, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are directed to a redundant system anda redundancy method.

BACKGROUND

A variety of nodes including a Web server, an application server and aDB (DataBase) server are set up in a data center where each of thesenodes is usually made redundant to be prepared for an accident or afailure.

There is known a redundancy technique, for example, in which a primarynode and a secondary node are provided as nodes in the data center sothat, when the primary node fails, the secondary node instead takes overprocessing of the failed primary node and continues the processing.

There is also known a technique in which a secondary data centerprovided as a backup of a primary data center takes over processingexecuted in the primary data center and continues the processing, whenthe primary data center is damaged.

Patent Literature 1: Japanese Laid-open Patent Publication No.2008-134986

When the secondary data center takes over the processing as asubstitution for the primary data center, a data lost incidental to theswitchover of the data center needs to be prevented. The switchover tothe secondary data center is needed for different reasons at differenttimes and, in a mode where switchover control is executed in the sameway regardless of the reason for the switchover, more data lost maypossibly occur.

SUMMARY

According to an aspect of the embodiments, a redundant system includes:a primary system including: a first node; and a second node thatacquires data update information generated according to a data updateperformed in the first node through a first intra-system transfer path;and a secondary system including: a third node that acquires data updateinformation generated according to a data update performed in the firstnode through a first inter-system transfer path set between the firstnode and the third node; and a fourth node that acquires the data updateinformation acquired by the second node through a second inter-systemtransfer path set between the second node and the fourth node, andacquire the data update information acquired by the third node through asecond intra-system transfer path. The fourth node includes: a processorthat executes a process including: when the secondary system in place ofthe primary system operates, executing first takeover processing orsecond takeover processing, the first takeover processing taking overthe primary system on the basis of the data update information acquiredfrom either the second inter-system transfer path or the secondintra-system transfer path, and the second takeover processing takingover the primary system on the basis of both the data update informationacquired from the second inter-system transfer path and the data updateinformation acquired from the second intra-system transfer path.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of an overall configurationof a redundancy system according to a first embodiment;

FIG. 2 is a functional block diagram illustrating a functionalconfiguration of each node in a primary center;

FIG. 3 is a diagram illustrating an example of a user log;

FIG. 4 is a diagram illustrating an example of a control log;

FIG. 5 is a diagram illustrating an example of a recovery point log;

FIG. 6 is a diagram illustrating an example of an update filetransmitted in inter-system communication;

FIG. 7 is a functional block diagram illustrating a functionalconfiguration of each node in a secondary center;

FIG. 8 is a flowchart illustrating the flowof notification processingperformed from a primary master node to a primary mirror node;

FIG. 9 is a flowchart illustrating the flow of notification processingperformed from the primary master node to a secondary master node;

FIG. 10 is a flowchart illustrating the flow of update processingperformed by the primary mirror node;

FIG. 11 is a flowchart illustrating the flow of notification processingperformed by the primary mirror node;

FIG. 12 is a flowchart illustrating the flow of update and notificationprocessing performed by the secondary master node;

FIG. 13 is a flowchart illustrating the flow of update processingperformed by a secondary mirror node;

FIG. 14 is a diagram illustrating a first example of system switchoverof the redundancy system;

FIG. 15 is a diagram illustrating a second example of the systemswitchover of the redundancy system;

FIG. 16 is a flowchart illustrating the flow of system switchoverprocessing performed by the secondary mirror node; and

FIG. 17 is a diagram illustrating an example of a hardwareconfiguration.

DESCRIPTION OF EMBODIMENTS

Preferred embodiments will be explained with reference to accompanyingdrawings. Note that the present invention is not to be limited by theseembodiments.

[a] First Embodiment Example of Overall Configuration

FIG. 1 is a diagram illustrating an example of an overall configurationof a redundancy system according to a first embodiment. As illustratedin FIG. 1, this system is a redundancy system in which a data center ismirrored by realizing a DB quadruplication function, i.e., a DBquad-redundancy function, the system including a primary center 1 and asecondary center 5 each being a data center.

The primary center 1 is a data center including a primary master node 10and a primary mirror node 20, and has a redundant configuration in whicha DB is mirrored. Likewise, the secondary center 5 is a data centerincluding a secondary master node 50 and a secondary mirror node 60, andhas a redundant configuration in which a DB is mirrored. The secondarycenter 5 functions as a backup of the primary center 1. Note that eachnode is an example of a DB server, a storage system and the like.

The primary master node 10 is an example of a first node having acurrent DB 12 updated by an operation, and is started up as a primarynode in normal operation. The primary master node 10 updates the currentDB 12 by using a business application or the like and extracts updateinformation indicating a difference before and after the update. Theprimary master node 10 transmits, to the primary mirror node 20, anupdate log indicating the updated content in synchronization with theupdate of the current DB 12, for example. Moreover, the primary masternode 10 generates an update file formed of a plurality of update logsand transmits the file to the secondary master node 50 at apredetermined interval.

The primary mirror node 20 is an example of a second node having astandby DB 22 updated in synchronization with the current DB 12, andfunctions as a backup of the primary master node 10 in normal operation.The primary mirror node 20 receives, through a first intra-systemtransfer path, the update log that is the update information generatedin accordance with a data update in the primary master node 10. Theprimary mirror node 20 then uses the update log received and updates thestandby DB 22. The primary mirror node 20 thereafter generates an updatefile formed of the update log received from the primary master node 10and transmits the file to the secondary mirror node 60 at apredetermined interval.

The secondary master node 50 is an example of a third node having astandby DB 52 which stores information equivalent to that stored in thecurrent DB 12, and functions as a master node of a secondary system innormal operation as a measure to deal with an accident or the like inthe primary center 1. A first inter-system transfer path is set betweenthe secondary master node 50 and the primary master node 10 so that thesecondary master node acquires, through the first inter-system transferpath, a piece of update information generated in accordance with thedata update in the primary master node 10.

Upon receiving the update file as the update information from theprimary master node 10, for example, the secondary master node 50extracts an update log from the update file received and uses eachupdate log extracted to update the standby DB 52. The secondary masternode 50 thereafter generates an update file formed of a plurality ofupdate logs received from the primary master node 10 and transmits thefile to the secondary mirror node 60 at a predetermined interval.

The secondary mirror node 60 is an example of a fourth node having astandby DB 62 which stores information equivalent to that stored in thecurrent DB 12, and functions as a mirror node of the secondary system innormal operation as a measure to deal with an accident or the like inthe primary center 1. A second inter-system transfer path is set betweenthe secondary mirror node 60 and the primary mirror node 20 so that thesecondary mirror node acquires, through the second inter-system transferpath, a piece of update information of the data acquired by the primarymirror node 20. The secondary mirror node 60 further acquires, through asecond intra-system transfer path, a piece of update information of thedata acquired by the secondary master node 50.

The secondary mirror node 60 receives the update file as the updateinformation from the primary mirror node 20 and receives the update logfrom the secondary master node 50, for example. The secondary mirrornode 60 then uses any of the update information received and updates thestandby DB 62.

When the secondary center 5 is operated as a substitution for theprimary center 1 in the aforementioned state, the secondary mirror node60 can execute both first takeover processing and second takeoverprocessing, the first takeover processing taking over the primary center1 on the basis of data update information acquired from either thesecond inter-system transfer path or the second intra-system transferpath, and the second takeover processing taking over the primary center1 on the basis of both the data update information acquired from thesecond inter-system transfer path and the data update informationacquired from the second intra-system transfer path.

That is, the secondary mirror node 60 in the DB quadruplication systemreceives the update information of the current DB 12 from the primarymaster node 10 from two systems, the primary mirror node 20 and thesecondary master node 50. When the primary system is under maintenance,the secondary mirror node 60 uses an arbitrary piece of updateinformation between the update information received from the two systemsto update the standby DB 62 and then executes system switchover. Whenthe primary system is down, on the other hand, the secondary mirror node60 uses the latest of the update information received from the twosystems to update the standby DB 62 and then executes the systemswitchover.

As a result, at the time of the system switchover, the switchover can beexecuted while preventing the data lost.

Functional Configuration of Each Node

Next, a functional configuration of each node illustrated in FIG. 1 willbe described. Here, while the functional configuration in a stateillustrated in FIG. 1 will be described as an example, each node canalso have the same functional configuration.

Functional Configuration of Primary Center

FIG. 2 is a functional block diagram illustrating a functionalconfiguration of each node in the primary center. The primary masternode 10 and the primary mirror node 20 included in the primary center 1will be described below.

Functional Configuration of Primary Master Node

As illustrated in FIG. 2, the primary master node 10 includes acommunication control unit 11, a DB 12, and a control unit 13.

The communication control unit 11 is a processor that controlscommunication between the primary master node and each of the primarymirror node 20 and the secondary master node 50, and is a networkinterface, for example. The communication control unit 11 constructs thefirst intra-system transfer path communicating with the primary mirrornode 20 and the first inter-system transfer path communicating with thesecondary master node 50, for example.

The DB 12 is a database storing operation information or the like andcorresponds to the current DB 12 illustrated in FIG. 1. The DB 12 isupdated by an operation. The DB 12 is provided in a storage such as ahard disk. The DB 12 corresponds to the active DB 12.

The control unit 13 is an example of a processor that controlsprocessing of the entire primary master node 10. The control unit 13executes a function that realizes a DB redundancy system of the primarymaster node 10 and the primary mirror node 20 as well as a function thatrealizes a DB redundancy system of the primary master node 10 and thesecondary master node 50.

That is, the control unit 13 executes an application realizing a DBduplication function within the primary center 1 and an applicationrealizing the DB quadruplication function across the centers. Whenperforming maintenance of the node, the control unit 13 shifts to amaintenance mode or the like after transmitting all the updateinformation already updated in the DB 12 to both the primary mirror node20 and the secondary master node 50. Note that the control unit 13 canalso notify each node about the shift to the maintenance mode throughthe same path as the path through which the update information istransmitted.

The control unit 13 includes a DB update unit 14, an intra-centernotification unit 15, an insertion unit 16, and an inter-centernotification unit 17. Each of these processing units is an example of anelectronic circuit included in the processor or processing executed bythe processor.

The DB update unit 14 is a processing unit that updates the DB 12. TheDB update unit 14 updates data stored in the DB 12 with the execution ofan application, for example.

The intra-center notification unit 15 is a processing unit thattransmits update information of the DB 12 to the primary mirror node 20within the same system in synchronization with the update of the DB 12.Specifically, the intra-center notification unit 15 extracts adifference in information before and after the update once the DB 12 isupdated. The intra-center notification unit 15 then uses the firstintra-system transfer path and transmits, to the primary mirror node 20,an update log indicating difference information as the updateinformation.

An example of the update log will be described below. FIG. 3 is adiagram illustrating an example of a user log, while FIG. 4 is a diagramillustrating an example of a control log. As illustrated in FIG. 3, theuser log being an example of the update log indicates the updateinformation of the DB and is formed of “header, user log display,variable length portion, variable length portion 2, and BC keyinformation”.

Information indicating the update log as well as a date and time createdare set to the “header”, while information indicating that it is theuser log is set to the “user log display”. Set to the “variable lengthportion” and the “variable length portion 2” is a piece of informationindicating the content of update of the DB such as a specific recordlocation, data before and after the update, and the differenceinformation. Information on the DB duplication between the primarymaster node 10 and the primary mirror node 20 such as checksuminformation and a serial number of the log are set to the “BC keyinformation”.

As illustrated in FIG. 4, the control log being an example of the updatelog indicates control processing such as rollback processing performedon the DB and is formed of “header, control log display, and COMMITspecification”. Information indicating the update log as well as a dateand time created are set to the “header”, while information indicatingthat it is the control log is set to the “control log display”.Information indicating specific control processing such as transactioninformation is set to the “COMMIT specification”.

When the DB 12 is updated, the intra-center notification unit 15generates the update log such as the user log or the control log byusing the updated information, as described above. The intra-centernotification unit 15 then transmits the generated update log to theprimary mirror node 20. The intra-center notification unit 15 furthernotifies the inter-center notification unit 17 of the generated updatelog. That is, the intra-center notification unit 15 notifies of theupdate information of the DB in synchronization with the update of theDB 12 within the same center.

When the update information generated in accordance with the update ofdata in the DB of the primary master node 10 is transmitted to theprimary mirror node 20 and the secondary master node 50, the insertionunit 16 inserts delimitation information, indicating the boundary of oneor a plurality of update processing units, to the data transmitted tothe primary mirror node and the secondary master node.

Specifically, the insertion unit 16 periodically generates a check pointwhich is a piece of determination information common to each node anddetermines an arrival state of the update log. The insertion unit 16then transmits the periodically generated check point to the primarymirror node 20 and notifies the inter-center notification unit 17. Notethat the check point in this case adopts a recovery point log that is anexample of the update log. Moreover, the time interval at which thecheck point is generated can be set to five seconds, for example, butmay be changed as appropriate.

FIG. 5 is a diagram illustrating an example of the recovery point log.As illustrated in FIG. 5, the recovery point log is a log indicating thecheck point which determines the arrival state of the update log and isformed of “header, control log display, and RP information”. Informationindicating the update log as well as a date and time created are set tothe “header”, while information indicating that it is the recovery pointlog is set to the “control log display”. The “RP information” is a pieceof information identifying the recovery point and includes an“identifier” and a “serial number”. The “identifier” is a piece ofinformation identifying that it is the check point information of the DBquadruplication function, while the “serial number” is a 23-byte fixedpositive number that is unique in the DB quadruplication system, where alarger number indicates a newer log, for example.

The inter-center notification unit 17 is a processing unit that putstogether the update information of the DB 12 to be periodicallytransmitted to the secondary master node 50 in a different system.Specifically, the inter-center notification unit 17 generates an updatefile in which the update log acquired from the intra-center notificationunit 15 and the recovery point log acquired from the insertion unit 16are put together in chronological order at an interval of 10 seconds,for example. The inter-center notification unit 17 then uses the firstinter-system transfer path and transmits the update file to thesecondary master node 50. That is, the inter-center notification unit 17generates the update information and performs the notificationasynchronously with the update of the DB 12 across different centers,the update information including the update information of the DB andthe check point that are periodically put together.

FIG. 6 is a diagram illustrating an example of the update filetransmitted in inter-system communication. As illustrated in FIG. 6, theupdate file is formed of the update log and the recovery point log. FIG.6 illustrates the example where an update log 1, an update log 2, and arecovery point log 1 included in the update file are generated in thisorder. Note that the update log 1 and update log 2 correspond to theaforementioned user log and control log, while the recovery point log 1corresponds to the aforementioned recovery point log.

Functional Configuration of Primary Mirror Node

As illustrated in FIG. 2, the primary mirror node 20 includes acommunication control unit 21, a DB 22, and a control unit 23.

The communication control unit 21 is a processor that controlscommunication between the primary mirror node and each of the primarymaster node 10 and the secondary mirror node 60, and is a networkinterface, for example. The communication control unit 21 constructs thefirst intra-system transfer path communicating with the primary masternode 10 and the second inter-system transfer path communicating with thesecondary mirror node 60, for example.

The DB 22 is a database storing operation information or the likesimilar to that stored in the DB 12 of the primary master node 10, andcorresponds to the standby DB 22 illustrated in FIG. 1. The DB 22 isupdated in synchronization with the DB 12. Note that the DB 22 isprovided in a storage such as a hard disk. The DB 22 corresponds to thestandby DB 22.

The control unit 23 is an example of a processor that controlsprocessing of the entire primary mirror node 20. The control unit 23executes a function that realizes the DB redundancy system of theprimary master node 10 and the primary mirror node 20 as well as afunction that realizes a DB redundancy system of the primary mirror node20 and the secondary mirror node 60. That is, the control unit 23executes an application realizing the DB duplication function within theprimary center 1 and an application realizing the DB quadruplicationfunction across the centers.

Note that, when performing maintenance of the node, the control unit 23shifts to a maintenance mode or the like after transmitting all theupdate information received from the primary master node 10 to thesecondary mirror node 60. The control unit 23 can also notify each nodeabout the shift to the maintenance mode through the same path as thepath through which the update information is transmitted.

The control unit 23 includes a reception unit 24, a DB update unit 25,and an inter-center notification unit 26. Each of these processing unitsis an example of an electronic circuit included in the processor orprocessing executed by the processor.

The reception unit 24 is a processing unit that receives the updateinformation of the DB 12 from the primary master node 10. Specifically,the reception unit 24 uses the first intra-system transfer path toreceive an update log in synchronization with the update of the DB 12 inthe primary master node 10 and notifies the DB update unit 25 and theinter-center notification unit 26. The reception unit 24 also notifiesthe inter-center notification unit 26 upon receiving the recovery pointlog.

The DB update unit 25 is a processing unit that uses the updateinformation of data notified by the primary master node 10 and updatesthe DB 22. The DB update unit 25 extracts a record to be updated orupdated data from the variable length portion of the update log receivedand then updates the DB 22 according to the extracted information, forexample. The DB update unit 25 updates the DB 22 every time the updatelog is received. As a result, the DB 22 can be synchronized with the DB12 of the primary master node 10 and function as a mirroring DB.

The inter-center notification unit 26 is a processing unit that putstogether the update information of the DB 22 to be periodicallytransmitted to the secondary mirror node 60 in a different system.Specifically, the inter-center notification unit 26 generates an updatefile in which the update log and recovery point log received from theprimary master node 10 are put together in chronological order at aninterval of 10 seconds, for example. The inter-center notification unit26 then uses the second inter-system transfer path to transmit theupdate file to the secondary mirror node 60. The inter-centernotification unit 26 generates the update file illustrated in FIG. 6 andtransmits the file to the secondary mirror node 60, for example.

Functional Configuration of Secondary Center

FIG. 7 is a functional block diagram illustrating a functionalconfiguration of each node in the secondary center. The secondary masternode 50 and the secondary mirror node 60 included in the secondarycenter 5 will be described below.

Functional Configuration of Secondary Master Node

As illustrated in FIG. 7, the secondary master node 50 includes acommunication control unit 51, a DB 52, and a control unit 53.

The communication control unit 51 is a processor that controlscommunication between the secondary master node and each of the primarymaster node 10 and the secondary mirror node 60, and is a networkinterface, for example. The communication control unit 51 constructs thefirst inter-system transfer path communicating with the primary masternode 10 and the second intra-system transfer path communicating with thesecondary mirror node 60, for example.

The DB 52 is a database storing operation information or the like andcorresponds to the standby DB 52 illustrated in FIG. 1. The DB 52 isupdated asynchronously with the update of the DB 12 by the updateinformation notified from the primary master node 10. Note that the DB52 is provided in a storage such as a hard disk. The DB 52 correspondsto the standby DB 52.

The control unit 53 is an example of a processor that controlsprocessing of the entire secondary master node 50. The control unit 53executes an application realizing the entire DB quadruplication systemacross the centers illustrated in FIG. 1 and an application realizingthe DB duplication function within the secondary center 5.

The control unit 53 includes a reception unit 54, a DB update unit 55,an intra-center notification unit 56, and a switchover control unit 57.Each of these processing units is an example of an electronic circuitincluded in the processor or processing executed by the processor.

The reception unit 54 is a processing unit that receives the updateinformation of the DB 12 from the primary master node 10. Specifically,the reception unit 54 uses the first inter-system transfer path toreceive the update file formed of the update log at a predeterminedinterval. The reception unit 54 then outputs the received update file tothe DB update unit 55.

The DB update unit 55 is a processing unit that updates the DB 52according to the update information of data notified by the primarymaster node 10. The DB update unit 55 extracts various logs included inthe update file that is received by the reception unit 54, for example.

The DB update unit 55 then identifies the user log and control log fromamong the extracted logs. After that, the DB update unit 55 reflects thedata update identified by the user log or the control log to the DB 52in the chronological order the logs are generated. The DB update unit 55further outputs, to the intra-center notification unit 56, each of thevarious logs extracted from the update file in the chronological order.

Upon receiving the update file illustrated in FIG. 6, for example, theDB update unit 55 extracts the update log 1, the update log 2, and therecovery point log 1 from the update file. The DB update unit 55 firstreflects the data update identified by the update log 1 into the DB 52and then reflects the data update identified by the update log 2 intothe DB 52. On the other hand, the DB update unit 55 outputs theextracted update log 1, update log 2, and recovery point log 1 to theintra-center notification unit 56.

The intra-center notification unit 56 is a processing unit thattransmits the update information of the data reflected in the DB 52 tothe secondary mirror node 60. Specifically, the intra-centernotification unit 56 uses the second intra-system transfer path andtransmits, to the secondary mirror node 60, the update log and therecovery point log received from the primary master node 10 in thechronological order each log is generated.

In the aforementioned example, the intra-center notification unit 56receives the update log 1, the update log 2, and the recovery point log1 in this order from the DB update unit 55. Then, the intra-centernotification unit 56 transmits to the secondary mirror node 60 theupdate log 1 first, followed by the update log 2 and lastly the recoverypoint log 1.

The switchover control unit 57 is a processing unit that executes systemswitchover or the like according to shutdown information of a nodereceived from a manager terminal or another node that is notillustrated. When the primary center 1 is shut down due to maintenance,for example, the switchover control unit 57 promotes the secondarymaster node to a primary master node after receiving all the updateinformation from the primary master node 10 and reflecting theinformation to the DB 52. That is, the switchover control unit 57notifies the DB update unit 55 about the promotion to the master node,while the DB update unit 55 updates the DB 52 by executing a businessapplication or the like.

Moreover, when receiving an instruction to be demoted to a mirror nodefrom the secondary mirror node 60 following the system down of theprimary center 1, the switchover control unit 57 demotes the secondarymaster node to a primary mirror node. That is, the switchover controlunit 57 instructs the intra-center notification unit 56 to stoptransmission processing performed through the second intra-systemtransfer path, and instructs the DB update unit 55 to start updateprocessing as a secondary mirror node. As a result, the DB update unit55 receives the update information from the secondary mirror node 60 andupdates the DB 52.

Functional Configuration of Secondary Mirror Node

As illustrated in FIG. 7, the secondary mirror node 60 includes acommunication control unit 61, a DB 62, a buffer 63, and a control unit64.

The communication control unit 61 is a processor that controlscommunication between the secondary mirror node and each of the primarymirror node 20 and the secondary master node 50, and is a networkinterface, for example. The communication control unit 61 constructs thesecond inter-system transfer path communicating with the primary mirrornode 20 and the second intra-system transfer path communicating with thesecondary master node 50, for example.

The DB 62 is a database storing operation information or the like andcorresponds to the standby DB 62 illustrated in FIG. 1. The DB 62 isupdated by the update information notified from the primary master node10, asynchronously with the update of the DB 12 in the primary masternode 10. On the other hand, the DB 62 is updated in synchronization withthe update of the DB 52 in the secondary master node 50. Note that theDB 62 is provided in a storage such as a hard disk. The DB 62corresponds to the standby DB 62.

The buffer 63 is a storage area temporarily storing the updateinformation received from the primary mirror node 20 by using the secondinter-system transfer path and the update information received from thesecondary master node 50 by using the second intra-system transfer path.Note that the buffer 63 is provided in a storage such as a hard disk ora memory.

The control unit 64 is an example of a processor that controlsprocessing of the entire secondary mirror node 60. The control unit 64executes an application realizing the entire DB quadruplication systemacross the centers illustrated in FIG. 1 and an application realizingthe DB duplication function within the secondary center 5.

The control unit 64 includes an intra-center reception unit 65, aninter-center reception unit 66, a DB update unit 67, and a switchovercontrol unit 68. Each of these processing units is an example of anelectronic circuit included in the processor or processing executed bythe processor.

The intra-center reception unit 65 is a processing unit that receivesthe update information of data from the secondary master node 50.Specifically, the intra-center reception unit 65 uses the secondintra-system transfer path to receive the update log and the recoverypoint log from the secondary master node 50 and stores the logs into thebuffer 63 in the chronological order of the date and time each log isgenerated. In the aforementioned example, the intra-center receptionunit 65 receives the update log 1, the update log 2, and the recoverypoint log 1 in this order from the secondary master node 50 and storesthe logs into the buffer 63 in the order the logs are received.

The inter-center reception unit 66 is a processing unit that receivesthe update information of data from the primary mirror node 20.Specifically, the inter-center reception unit 66 uses the secondinter-system transfer path to receive the update file formed of theupdate log and the recovery point log from the primary mirror node 20.The inter-center reception unit 66 then extracts the various logs fromthe update file and stores the logs into the buffer 63 in the order ofthe data and time each log is generated.

The DB update unit 67 is a processing unit that uses the updateinformation received and updates the DB 62. Specifically, the DB updateunit 67 reads from the buffer 63 the update log received by theintra-center reception unit 65 in the order each update log isgenerated, and successively updates the DB 62 according to the updatelog being read.

Moreover, the DB update unit 67 selects and executes DB updateprocessing in the system switchover by an instruction from theswitchover control unit 68 to be described later. When notified from theswitchover control unit 68 that the maintenance of the primary center 1is started, for example, the DB update unit 67 updates the DB 62 byusing either the update information received by the intra-centerreception unit 65 or the update information received by the inter-centerreception unit 66. After that, the DB update unit 67 executes the systemswitchover by which the secondary mirror node is operated as a mirrornode in the primary system. That is, the DB update unit 67 updates theDB 62 by using the update information received by the intra-centerreception unit 65 after the system switchover is started.

When notified of the system down of the primary center 1 from theswitchover control unit 68, on the other hand, the DB update unit 67selects the latest update information between the update informationreceived by the intra-center reception unit 65 and the updateinformation received by the inter-center reception unit 66, and reflectsthe information to the DB 62. The DB update unit 67 determines whichupdate information is the latest on the basis of the serial number orthe like within the recovery point log, for example.

When the update information received by the intra-center reception unit65 is determined to be the latest, the DB update unit 67 executes thesystem switchover by which the secondary mirror node is operated as amirror node in the primary system. That is, the DB update unit 67updates the DB 62 by using the update information received by theintra-center reception unit 65 after the system switchover is started.

When the update information received by the inter-center reception unit66 is determined to be the latest, on the other hand, the DB update unit67 executes the system switchover by which the secondary mirror node isoperated as a master node in the primary system. That is, the DB updateunit 67 transmits a demotion instruction to the secondary master node 50to be operated as a mirror node in the primary system. The DB updateunit 67 further updates the DB 62 by executing a business application orthe like and transmits the update information of the DB 62 to thesecondary master node 50 by using the second intra-system transfer path.

The switchover control unit 68 is a processing unit that executes thesystem switchover or the like according to shutdown information of anode received from a manager terminal or another node that is notillustrated. When the primary center 1 is shut down due to maintenance,for example, the switchover control unit 68 notifies the DB update unit67 that the maintenance of the primary center 1 is started. When theprimary center 1 experiences the system down, the switchover controlunit 68 notifies the DB update unit 67 about the system down of theprimary center 1.

Flow of Processing Related to DB Update

Now, DB update processing and update information notification processingexecuted by each node will be described. Note that while there will bedescribed an example where the check point (recovery point log) isgenerated after updating the DB, it is not limited to such example. Theprocessing of updating the DB and the processing of generating the checkpoint can be executed concurrently or in separate flowcharts, forexample.

Notification Processing from Primary Master Node 10 to Primary MirrorNode 20

FIG. 8 is a flowchart illustrating the flow of notification processingperformed from the primary master node to the primary mirror node.

As illustrated in FIG. 8, the DB update unit 14 of the primary masternode 10 updates the DB 12 (Step S102) once an update to the DB 12 isgenerated (Step S101: Yes). Next, the intra-center notification unit 15extracts a difference in the updated DB 12 before and after the update(Step S103), and generates the update log and transmits it to theprimary mirror node 20 (Step S104).

On the other hand, at a timing to generate the check point (Step S105:Yes), the insertion unit 16 generates the recovery point log containingthe serial number of the check point (Step S106) and transmits the logto the primary mirror node 20 (Step S107). Note that the processingreturns to Step S101 when it is not the timing to generate the checkpoint (Step S105: No), in which case the processing from Step S101onward is executed.

Notification Processing from Primary Master Node 10 to Secondary MasterNode 50

FIG. 9 is a flowchart illustrating the flow of notification processingperformed from the primary master node to the secondary master node.

As illustrated in FIG. 9, the DB update unit 14 of the primary masternode 10 updates the DB 12 (Step S202) once an update to the DB 12 isgenerated (Step S201: Yes). The inter-center notification unit 17thereafter extracts and accumulates a difference in the updated DB 12before and after the update (Step S203).

On the other hand, at a timing to generate the check point (Step S204:Yes), the insertion unit 16 generates and accumulates the recovery pointlog containing the serial number of the check point (Step S205). Notethat Step S205 is not executed when it is not the timing to generate thecheck point (Step S204: No), in which case the processing in Step S206is executed.

At a timing to notify the secondary master node 50 (Step S206: Yes), theinter-center notification unit 17 generates the update file in which theaccumulated update log and recovery point log are written in the ordereach log is generated (Step S207). The inter-center notification unit 17then transmits the generated update file to the secondary master node 50(Step S208). Note that the processing returns to Step S201 when it isnot the timing to generate the check point (Step S206: No), in whichcase the processing from Step S201 onward is executed.

Update Processing of Primary Mirror Node 20

FIG. 10 is a flowchart illustrating the flow of update processingperformed by the primary mirror node. Upon receiving information fromthe primary master node (Step S301: Yes), as illustrated in FIG. 10, thereception unit 24 of the primary mirror node 20 determines whether ornot the received information is the update log (Step S302).

Next, the DB update unit 25 updates the DB 22 according to the updatelog received (Step S303) when the received information is the update log(Step S302: Yes), and accumulates the update log used in the update(Step S304).

When the received information is not the update log but the recoverypoint log (Step S302: No), on the other hand, the DB update unit 25accumulates the received recovery point log in a storage unit or thelike (Step S305).

Notification Processing of Primary Mirror Node 20

FIG. 11 is a flowchart illustrating the flow of notification processingperformed by the primary mirror node. At a notification timing (StepS401: Yes), the inter-center notification unit 26 of the primary mirrornode 20 reads the accumulated update log and recovery point log (StepS402), as illustrated in FIG. 11.

The inter-center notification unit 26 thereafter generates the updatefile by rearranging each log being read in the order each log isgenerated (Step S403) and transmits the generated update file to thesecondary mirror node 60 in the secondary center 5 by using the firstinter-system transfer path (Step S404).

Processing Performed by Secondary Master Node 50

FIG. 12 is a flowchart illustrating the flow of update and notificationprocessing performed by the secondary master node. When the update fileis received (Step S501: Yes), as illustrated in FIG. 12, the DB updateunit 55 of the secondary master node 50 extracts each log included inthe update file (Step S502).

Next, the DB update unit 55 successively reflects in the DB 52 theupdate log, from among the acquired logs, in the chronological ordereach update log is generated (Step S503) and repeats the processinguntil all the extracted update logs are reflected (Step S504: No).

When all the extracted update logs are reflected (Step S504: Yes), theintra-center notification unit 56 successively transmits to thesecondary mirror node 60 the update log and recovery point log acquiredfrom the update file in the chronological order each log is generated(Step S505). The intra-center notification unit 56 repeats theprocessing until all the logs acquired from the update file aretransmitted (Step S506: No) and, when all the logs are transmitted (StepS506: Yes), the processing from Step S501 onward is repeated byreturning thereto.

Processing Performed by Secondary Mirror Node 60

FIG. 13 is a flowchart illustrating the flow of update processingperformed by the secondary mirror node. When the intra-center receptionunit 65 receives information from the secondary master node 50 (StepS601: Yes), as illustrated in FIG. 13, the DB update unit 67 of thesecondary mirror node 60 determines whether or not the receivedinformation is the update log (Step S602).

The DB update unit 67 then reflects the received update log into the DB62 and updates the DB 62 (Step S603) when the received information isthe update log (Step S602: Yes). When the received information is notthe update log but the recovery point log (Step S602: No), on the otherhand, the DB update unit 67 accumulates the recovery point log in thebuffer 63 or the like (Step S604).

Also, when the inter-center reception unit 66 receives the update filenot from the secondary master node 50 but the primary mirror node 20 inStep S601 (Step S601: No, Step S605: Yes), the log is extracted from theupdate file (Step S606). The inter-center reception unit 66 thenaccumulates the acquired update log and recovery point log into thebuffer 63 in the chronological order each log is generated (Step S607).

After that, the DB update unit 67 compares the recovery point logsstored in the buffer 63 at a regular interval (Step S608). The DB updateunit 67 executes processing in Step S610 when the recovery point fromthe secondary master node 50 is the latest (Step S609: Yes). That is,the DB update unit 67 deletes from the buffer 63 the update log andrecovery point log received from the primary mirror node 20.

When the recovery point log from the primary mirror node 20 is thelatest (Step S609: No), on the other hand, the DB update unit 67 repeatsthe processing from Step S601 onward while keeping the update log andrecovery point log received from the primary mirror node 20 in thebuffer 63.

Example of System Switchover Pattern

Next, there will be described a specific example of system switchoverperformed in the redundancy system illustrated in FIG. 1. FIG. 14 is adiagram illustrating a first example of the system switchover of theredundancy system, and FIG. 15 is a diagram illustrating a secondexample of the system switchover of the redundancy system.

First Example of System Switchover

FIG. 14 is a diagram illustrating an example of the system switchoverperformed when the primary master node 10 and the primary mirror node 20are shut down intentionally in order for the primary center 1 to performmaintenance. Upon receiving an instruction to start the maintenance froma manager or the like, as illustrated in FIG. 14, the primary masternode 10 transmits the update information up to the latest one stored inthe current DB 12 to the primary mirror node 20 through the firstintra-system transfer path and to the secondary master node 50 throughthe first inter-system transfer path.

The primary master node 10 then shifts to the maintenance mode aftercompleting the transmission of the latest update information of thecurrent DB 12. As a result, the update information the secondary mirrornode 60 receives from the primary mirror node 20 through the secondinter-system transfer path and the update information the secondarymirror node receives from the secondary master node 50 through thesecond intra-system transfer path are both the latest updateinformation. The secondary mirror node 60 therefore uses either of theupdate information received through the two paths and updates thestandby DB 62. In this case, the secondary master node 50 is promoted toa primary master node, and the secondary mirror node 60 is promoted to aprimary mirror node.

Second Example of System Switchover

FIG. 15 is a diagram illustrating an example of the system switchoverperformed when the primary master node 10 and the primary mirror node 20are shut down abruptly due to the system down of the primary center 1.The primary master node 10 is shut down abruptly as illustrated in FIG.15. In this case, one is difficult to determine the update informationup to which point is transmitted to the primary mirror node 20 throughthe first intra-system transfer path and the update information up towhich point is transmitted to the secondary master node 50 through thefirst inter-system transfer path.

Accordingly, the secondary mirror node 60 identifies the newer piece ofupdate information between the update information received through thetwo paths, and updates the standby DB 62 by using the identified updateinformation. When the update information from the secondary master node50 is the latest, the secondary master node 50 is promoted to a primarymaster node, and the secondary mirror node 60 is promoted to a primarymirror node. When the update information from the primary mirror node 20is the latest, the secondary master node 50 is demoted to a primarymirror node, and the secondary mirror node 60 is promoted to a primarymaster node.

Flow Followed by Secondary Mirror Node in System Switchover

FIG. 16 is a flowchart illustrating the flow of system switchoverprocessing performed by the secondary mirror node. As illustrated inFIG. 16, the switchover control unit 68 of the secondary mirror node 60detects a shutdown instruction of the primary center 1 being the primarysystem (Step S701: Yes) and determines whether or not the reason for thesystem shutdown is maintenance (Step S702).

When the reason for the system shutdown is the maintenance (Step S702:Yes), the DB update unit 67 determines whether or not the updateinformation up to the latest one is received from each node (Step S703).The switchover control unit 68 waits for a predetermined period of timeafter receiving the shutdown instruction, for example. Alternatively,the switchover control unit 68 determines that the update information upto the latest one is received when a new piece of update information isnot received after receiving the shutdown instruction and waiting forthe predetermined period of time.

When the update information up to the latest one is received from eachnode (Step S703: Yes), the DB update unit 67 selects an arbitrary pieceof update information between the update information from the primarymirror node 20 and the update information from the secondary master node50 (Step S704), and updates the DB 62 (Step S705).

The DB update unit 67 thereafter receives the update information fromthe secondary master node 50 that is promoted to a master node in theprimary system (Step S706: Yes) and updates the DB 62 by using thereceived update information (Step S707). The processing from Step S706onward is then repeated.

When the reason for the system shutdown is not the maintenance (StepS702: No), on the other hand, the DB update unit 67 identifies thelatest update information between the update information from theprimary mirror node 20 and the update information from the secondarymaster node 50 (Step S708).

When the update information from the secondary master node 50 is thelatest (Step S709: Yes), the DB update unit 67 deletes from the buffer63 the update information transmitted from the primary mirror node 20(Step S710). The DB update unit 67 thereafter executes the processingfrom Step S706 onward.

When the update information from the primary mirror node 20 is thelatest (Step S709: No), on the other hand, the DB update unit 67 updatesthe DB 62 by using the update information from the primary mirror node(Step S711).

Successively, the DB update unit 67 promotes the secondary mirror nodeto a master node in the primary system (Step S712) and transmits ademotion instruction to the secondary master node 50 to be demoted to amirror node in the primary system (Step S713).

When an update to the DB is generated thereafter (Step S714: Yes), theDB update unit 67 updates the DB 62 (Step S715). Next, the DB updateunit 67 extracts a difference in the updated DB 62 before and after theupdate, generates the update log and transmits it to the secondarymaster node 50 (Step S716). The processing from Step S714 onward is thenrepeated.

The primary master node 10 shifts to the maintenance after transmittingall the update information of the current DB 12 when performing themaintenance of the primary system, as described above, so that all theupdate information the secondary mirror node 60 receives through the twopaths is the latest update information. When the primary system is down,on the other hand, the primary master node 10 is shut down withoutwaiting for the update information to be completely transmitted, so thatthere may be a time lag in the update information received by thesecondary mirror node 60 through the two paths.

As a result, a method of selecting the update information can beswitched between the time of maintenance and the system down when thesystem switchover occurs, whereby the switchover can be performed athigh speed while preventing the data lost.

Moreover, the system switchover can be executed upon changing themaster/mirror relationship between the secondary mirror node 60 and thesecondary master node 50 depending on the state of the updateinformation already received by the secondary mirror node 60. As aresult, the nodes can be switched over in conjunction with each othersuch that the latest possible update information is used to continue theoperation. Moreover, the reliability of the system is improved since therisk of stopping the operation following the shutdown of the node can bereduced.

The primary master node 10 periodically transmits a check point whentransmitting the update information of the DB 12. The secondary mirrornode 60 receives the update information and check point through the twopaths. Accordingly, the secondary mirror node 60 can update the DB 62 byusing the update information received through the path with the newercheck point. The secondary mirror node 60 can thus prevent the data lostof the update information used to update the DB 62.

The primary master node 10 updates the DB 12 and then transmits theupdate log to the primary mirror node 20. As a result, the primarymirror node 20 can synchronize the state of the DB 22 with the DB 12.Moreover, the primary master node 10 periodically transmits, to thesecondary master node 50, the update file in which the logs updating theDB 12 are put together. The secondary master node 50 can thus update theDB 52 by decreasing a time lag from the update time of the DB 12.

[b] Second Embodiment

While there has been described the embodiment of the present invention,the present invention may be implemented in various different modesother than the aforementioned embodiment.

Check Point

While there has been described the example where the recovery point logincluding the serial number unique within the system is used as thecheck point in the aforementioned embodiment, a recovery point logincluding a date and time can be used as the check point as well. Inother words, various pieces of information can be used as long as aunique order such as an ascending order or a descending order can bespecified within the system.

System Shutdown

While there has been described the example where the node is shut downas a result of a failure or the like in the aforementioned embodiment,similar processing can be performed during maintenance work or the like.Moreover, there has been described the example where any of the nodes isshut down in the aforementioned embodiment, but the system shutdown isnot limited to what is described above. The switchover processing can beexecuted in the similar manner when periodic monitoring or a monitoringtool detects that the communication between the nodes is shut down, forexample.

The redundancy system can execute the processing similar to thatexecuted at the time of the system down described above even when thecommunication between the primary center 1 and the secondary center 5 iscut off, for example.

While there has been described the example where the secondary mirrornode 60 selects the latest update information between the updateinformation received through the two paths and reflects it to the DB 62at the time of the system down, it is not limited to such example. Thesecondary mirror node 60 can execute the system switchover afterreflecting each of the update information received through the two pathsinto the DB 62, for example. Moreover, upon receiving the promotioninstruction from the manager terminal or the like, the secondary mirrornode 60 reflects to the DB 62 the latest update information, namely theupdate information indicating a further progressed transaction, betweenthe update information from the primary mirror node 20 and the updateinformation from the secondary master node 50, and then transmits thedemotion instruction to the secondary master node 50. The secondarymirror node 60 can also start processing as a primary master node.

System

Moreover, all or a part of the processings described to be performedautomatically, among the processings described in the presentembodiment, can also be performed manually. On the other hand, all or apart of the processings described to be performed manually can also beperformed automatically by using a known method. In addition, theinformation including the procedure, the control procedure, the specificname and the various data and parameters provided in the description anddrawings above can be modified at will unless otherwise specified.

Moreover, each element of each device in the drawings illustrates afunctional concept and does not always physically configured asillustrated in the drawings. In other words, a specific mode ofdistribution or integration of each device is not limited to what isillustrated in the drawings. All or a part of the element can beconfigured while functionally or physically distributed/integrated by anarbitrary unit according to various loads and a use status. Furthermore,all or an arbitrary part of each processing function performed in eachdevice can be realized by a CPU and a program analyzed/executed by theCPU, or realized as hardware employing wired logic.

Hardware

FIG. 17 is a diagram illustrating an example of a hardwareconfiguration. Each node illustrated in FIG. 1 has a similar hardwareconfiguration so that, in this case, the primary master node 10 isdescribed as an example.

As illustrated in FIG. 17, the primary master node 10 includes an HDD(Hard Disk Drive) 10 a, a communication interface 10 b, a memory 10 c,and a CPU (Central Processing Unit) 10 d. Each unit illustrated in FIG.17 is mutually connected via a bus or the like. Note that the hardwareillustrated herein is provided as an example where another hardware suchas a graphic interface or a mouse may be included.

The HDD 10 a stores a DB and a program that actuates the functionillustrated in FIG. 2 and the like. The communication interface 10 b isan interface that controls communication with another device and is anetwork interface card, for example.

The CPU 10 d actuates a process executing each function illustrated inFIG. 2 and the like by reading, from the HDD 10 a or the like, theprogram that executes the processing similar to that performed by eachprocessing unit illustrated in FIG. 2 and the like and extracting theprogram in the memory 10 c.

That is, this process executes the function similar to that performed byeach processing unit included in the primary master node 10.Specifically, the CPU 10 d reads from the HDD 10 a or the like theprogram having a function similar to that of the DB update unit 14, theintra-center notification unit 15, the insertion unit 16, and theinter-center notification unit 17. The CPU 10 d then executes theprocess executing the processing similar to that performed by the DBupdate unit 14, the intra-center notification unit 15, the insertionunit 16, and the inter-center notification unit 17.

The primary master node 10 as a result operates as an informationprocessor that executes a redundancy creation method by reading andexecuting the program. Moreover, the primary master node 10 can realizethe function similar to that of the aforementioned embodiments byreading the program from a recording medium by a medium reading deviceand executing the program being read. Note that a program in anotherembodiment is not always executed by the primary master node 10. Thepresent invention can be similarly applied when the program is executedby another computer or server, or by cooperation between the computerand the server, for example.

According to one aspect, the switchover can be executed while preventingthe data lost.

All examples and conditional language recited herein are intended forpedagogical purposes of aiding the reader in understanding the inventionand the concepts contributed by the inventor to further the art, and arenot to be construed as limitations to such specifically recited examplesand conditions, nor does the organization of such examples in thespecification relate to a showing of the superiority and inferiority ofthe invention. Although the embodiments of the present invention havebeen described in detail, it should be understood that the variouschanges, substitutions, and alterations could be made hereto withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A redundant system comprising: a primary systemincluding: a first node; and a second node that acquires data updateinformation generated according to a data update performed in the firstnode through a first intra-system transfer path; and a secondary systemincluding: a third node that acquires data update information generatedaccording to a data update performed in the first node through a firstinter-system transfer path set between the first node and the thirdnode; and a fourth node that acquires the data update informationacquired by the second node through a second inter-system transfer pathset between the second node and the fourth node, and acquire the dataupdate information acquired by the third node through a secondintra-system transfer path, wherein, the fourth node includes: aprocessor that executes a process including: when the secondary systemin place of the primary system operates, executing first takeoverprocessing or second takeover processing, the first takeover processingtaking over the primary system on the basis of the data updateinformation acquired from either the second inter-system transfer pathor the second intra-system transfer path, and the second takeoverprocessing taking over the primary system on the basis of both the dataupdate information acquired from the second inter-system transfer pathand the data update information acquired from the second intra-systemtransfer path.
 2. The redundant system according to claim 1, wherein thesecond takeover processing compares the data update information acquiredfrom the second inter-system transfer path and the data updateinformation acquired from the second intra-system transfer path, andtakes over the primary system on the basis of one of the updateinformation including update information indicating a further progressedtransaction.
 3. The redundant system according to claim 1, wherein thefirst takeover processing does not switch master/mirror relationshipbetween the third node and the fourth node, while the second takeoverprocessing switches the master/mirror relationship between the thirdnode and the fourth node.
 4. The redundant system according to claim 1,wherein the first takeover processing is takeover processing associatedwith maintenance of the primary system, and the second takeoverprocessing is takeover processing associated with abnormal shutdown ofthe primary system.
 5. The redundant system according to claim 1,wherein the second takeover processing promotes the secondary system tothe primary system in response to completion of reflection of theacquired data update information to storage data in the fourth node. 6.A method for a redundant system including a primary system and asecondary system, the method comprising: transmitting data updateinformation generated according to a data update performed in a firstnode in the primary system, from the first node to a second node in theprimary system, through a first intra-system transfer path; transmittingdata update information generated according to a data update performedin the first node, from the first node to a third node in the secondarysystem, through a first inter-system transfer path set between the firstnode and the third node; transmitting the data update informationacquired by the second node, from the second node to a fourth node inthe secondary system, through a second inter-system transfer path setbetween the second node and the fourth node; transmitting the dataupdate information acquired by the third node, from the third node tothe fourth node, through a second intra-system transfer path; and whenthe secondary system in place of the primary system operates, executing,by the fourth node, first takeover processing or second takeoverprocessing, the first takeover processing taking over the primary systemon the basis of the data update information acquired from either thesecond inter-system transfer path or the second intra-system transferpath, and the second takeover processing taking over the primary systemon the basis of both the data update information acquired from thesecond inter-system transfer path and the data update informationacquired from the second intra-system transfer path.